This PR prototypes a new experimental Parquet repetition type, VECTOR, mainly for Arrow's FixedSizeList<T, N> as proposed in Option B here.

Today this is written through Parquet LIST, which means we pay repetition/definition-level costs for an inner shape that is already fixed. VECTOR stores the fixed length in the Parquet schema as vector_length and avoids the inner repetition level.

Writing FixedSizeList<T, N> as VECTOR is opt-in via ArrowWriterProperties::Builder::enable_experimental_vector_encoding() defaulting to LIST otherwise.

Implemented in this prototype

• Parquet schema/thrift addition of VECTOR
• Arrow schema conversion:
  • FixedSizeList -> VECTOR when explicitly enabled
  • VECTOR -> FixedSizeList
• required and nullable rows in VECTOR
• nullable primitive elements in VECTOR
• primitive type vectors
• limited composability proof of concept: FixedSizeList<struct<x: float, y: int32>, N>
• roundtrip/schema/path tests
• read/write/roundtrip benchmarks

Deferred for now:

• dictionary / non-PLAIN encodings
• statistics
• page index
• general nested VECTOR children (non-fixed will not be possible)

Compatibility

We expect non-VECTOR-aware readers to fail when encountering VECTOR repetition level.

Benchmark snapshot

Both VECTOR and LIST use identical WriterProperties (dictionary, statistics and page-index disabled, PLAIN encoding) — the only intended difference is enable_experimental_vector_encoding.

Numbers below are from a local release build on m4 mac, so directional only.

Required FixedSizeList<float, N>: FixedSizeList encoded as LIST vs VECTOR

All with FixedSizeList<float, {80,768,10k}>.

Realistic embedding row: (int64 id, timestamp ts, int32 category, float score, string label, FSL<float, N> embedding)